Object extracting method using motion picture

ABSTRACT

An object extracting method using a motion picture more accurately and rapidly extracts a specific object by using a difference image frame of a pair of still image frames obtained from a motion picture and color information which defines a color of the object. The object extracting method using a motion picture includes the steps of: obtaining a difference image frame by getting a pair of still image frames having a predetermined time interval from a motion picture and obtaining a color image frame which satisfies color information defining a color of a particular object from one of the still image frames; performing a grid processing of a logic image frame which is obtained from the difference image frame and the color image frame at a predetermined size and obtaining connected components using direction connection information and defining minimum areas each includes the connected components; comparing each of the minimum areas with predetermined conditions, thereby selecting the minimum areas which satisfy the conditions as object area candidates; and selecting and optimizing a largest object area candidate among the object area candidates.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to a technique of extracting aspecific object in a motion picture, and more particularly to an objectextracting method using motion pictures which more accurately andrapidly extracts an object using information of a moving object in amotion picture and color information which defines a color of theobject.

[0003] 2. Description of the Conventional Art

[0004] A motion picture is composed of image frames, each frame carriesinformation with respect to an object. Recently, a conventionaltechnique which extracts a specific object using the motion picture canbe divided into two ways; one is to extract the object using only colorinformation which defines a color thereof and the other is to extractthe object using information of motion changes of the object.

[0005] First, the conventional object extracting method using the colorinformation will be explained.

[0006] After obtaining a still image frame from a motion picture, apreprocessing step is performed, which generates a color histogram byconverting a negative image of the still image frame to a positiveimage.

[0007] Next, red, green and blue (RGB) color domains included in thehistogram are converted to hue saturation values (HSV), and colorinformation of the object such as skin-color pixels are obtained. To thecolor pixels, segmentation occurs through edge detection, hole filteringand gap filtering. Finally, the segments and a predetermined objectdomain are compared, thus the specific object is extracted.

[0008] However, because the conventional object extracting method usingonly the color information obtains one still image frame using the colorinformation which defines the color of the object and thus extracts theobject therefrom, the object may not be accurately obtained.Accordingly, the above object extracting method using the colorinformation requires a considerable operation time to extract the objectbecause RGB color space is converted to HSV color space with respect tothe inaccurate object domain.

[0009] A face-it method which is the other object extracting methodextracts a specific object only using motion information of the objecton a motion picture, without using color information thereof.Particularly, the face-it method which extracts a human face designatesan area in which there is a movement of a specific object as a facedomain candidate and carries out a grey image process for the domaincandidate, thereby obtaining information with respect to thecorresponding human face. Accordingly, the face-it which extracts thehuman face with insufficient information has difficulty of accuratelyextracting a human face.

SUMMARY OF THE INVENTION

[0010] Accordingly, an object of the present invention is to provide anobject extracting method using a motion picture which accurately andrapidly extracts an object using information of a moving object in amotion picture and color information which define a color of the object.

[0011] To achieve these and other advantages and in accordance with thepurpose of the present invention, as embodied and broadly described, anobject extracting method includes the steps of: obtaining a differenceimage frame by getting a pair of still image frames having apredetermined time difference from a motion picture and obtaining acolor image frame which satisfies color information defining a color ofa particular object from one of the still image frames; performing agrid processing of a logic image frame which is obtained from thedifference image frame and the color image frame at a predetermined sizeand obtaining connected components using direction connectioninformation and defining minimum areas each includes the connectedcomponents; comparing each of the minimum areas with predeterminedconditions, thereby selecting the minimum areas which satisfy theconditions as object area candidates; and selecting and optimizing alargest object area candidate among the object area candidates.

[0012] It is to be understood that both the foregoing generaldescription and the following detailed description are exemplary andexplanatory and are intended to provide and further explanation of theinvention as claimed.

DETAILED DESCRIPTION OF THE INVENTION

[0013] The accompanying drawings, which are included to provide afurther understanding of the invention and are incorporated in andconstitute a part of this specification, illustrate embodiments of theinvention and together with the description serve to explain theprinciples of the invention.

[0014] In the drawings:

[0015]FIG. 1 is a flowchart which illustrates an object extractingmethod using a motion picture according to the present invention;

[0016]FIG. 2 is a diagram illustrating an example of a connectedcomponent which is generated in accordance with 8-direction connectioninformation;

[0017]FIG. 3A is a picture which illustrates a first still image frameobtained from a motion picture;

[0018]FIG. 3B is a picture which illustrates a second still image frameobtained from a motion picture after a predetermined time elapses fromthe time at which the first still image frame is obtained;

[0019]FIG. 4 illustrates a difference image frame using the first stillimage frame and the second still image frame in FIGS. 3A and 3B;

[0020]FIG. 5 illustrates a skin color image frame obtained from thesecond still image frame of FIG. 3B;

[0021]FIG. 6 illustrates a logic image frame which is obtained by ANDingthe difference image frame of FIG. 4 and the skin color image frame ofFIG. 5;

[0022]FIG. 7 illustrates a grid image frame obtained by which a gridprocess is applied to the logic image frame of FIG. 6;

[0023]FIG. 8 illustrates minimum rectangles each includes a connectedcomponent; and

[0024]FIG. 9 is a picture which illustrates a human face extracted froma specific object in a motion picture according to the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

[0025] Now, the object extracting method using a motion pictureaccording to the present invention will be described in detail withreference to the accompanying drawings.

[0026]FIG. 1 is a flowchart which illustrates the object extractingmethod using a motion picture according to the present invention. In afirst step (S1), a first still image frame IMAGE (t) is obtained at atime (t) from a motion picture. After a predetermined time (Δt) elapsesfrom the time (t), a second still image frame IMAGE(t+Δt) is obtained ina second step (S2). In a third step (S3), a difference image frame whichhas information with respect to motion changes of an object is obtainedfrom the first still image frame IMAGE(t) and the second still imageframe IMAGE(t+Δt).

[0027] While, in a fourth step (S4) a color image frame which satisfiescolor information with respect to the object is obtained from the secondstill image frame IMAGE(t+Δt), Here, the color image frame can beobtained from the first still image frame IMAGE(t). The color imageframe which has been obtained from the first still image frame IMAGE(t)or the second still image frame IMAGE(t+Δt) outputs an identical resultin the object extracting method according to the present invention.

[0028] In a fifth step (S5) the color image frame and the differenceimage frame are ANDed, thus generating a logic image frame which has thecolor information and the motion change information of the object andthen a grid image frame is generated by performing a grid process withrespect to the logic image frame. Here, the grid process for the logicimage frame reduces operation capacity and time compared with which thelogic image frame is processed by the pixel. Specifically, the gridprocess divides the logic image frame into multiple grids, each has afixed size, and compares a predetermined value with a value representingpixels of the grid, and expresses a value of the grid which is largerthan the predetermined value as a binary grid image, thereby reducingthe operation capacity and time to process the logic image frame.

[0029] However, because each grid indicates a part of the object, thegrids which are gathered may have a shape similar to the object. Thus,in a sixth step (S6) using direction connection information it isdetermined whether the grids are connected with other grids, and ifconnected, the grids are defined as a connected component and thus thelogic image frame includes a plurality of connected components. Inaddition, in the sixth step (S6), minimum rectangles each includes eachof the connected components are obtained. Here, the minimum rectanglesare defined as a variable RECT[i] wherein i is an integer numberindicates a number of minimum rectangles. The minimum rectanglerepresents a candidate of a specific object to be extracted.

[0030] In a seventh step (S7), only each of the minimum rectangles, thecandidates of the specific object and minimum rectangles satisfyingconditions for the specific object. Here, the conditions which definethe object are as follows.

[0031] (1) A size of a variable RECT[i]>a threshold value of a size ofthe minimum rectangle

[0032] (2) A minimum value of a row/column ratio of the variableRECT[i]<the row/column ratio of the variable RECT[i]<a maximum value ofthe row/column ratio of the variable RECT[i]

[0033] (3) A density of the variable RECT[i]<a threshold value of adensity.

[0034] Here, the threshold value of the size of the minimum rectangleand the minimum and maximum values of the row/column ratio of thevariable RECT[i] are well known in the relevant field of the presentinvention, and in the variable RECT[i] of a minimum rectangle whichincludes connected components the density is a value of which the numberof grids in a row or a column of one of the connected components isdivided by a length of the row or the column thereof. While, the densityof the variable RECT[i] is a value of which the density of the connectedcomponents is divided by an area size of the variable RECT[i].

[0035] In an eighth step (S8), among the minimum rectangles the minimumrectangle having the maximum size is selected. Finally, in a ninth step(S9), the maximum sized rectangle is optimized to correspond to theobject. In addition, if an image considerably has noise, the ninth stepmay be performed first before the eighth step is carried out.

[0036]FIG. 2 is a diagram which illustrates an example of a connectedcomponent which is generated in accordance with 8-direction connectioninformation, which explains the sixth step (S6) of FIG. 1.

[0037] When each of G1, G2, G3 and G4 represents a grid and D1, D2, . .. ,D8 respectively indicate 8 directions, direction connectioninformation show a condition in which a grid is connected with othergrids in accordance with 4 directions or 8 directions thereof. That is,a grid G3 is connected with a grid G2 in the direction of D4 andconnected with a grid G4 in the direction of D8. Accordingly, the gridsG3, G2, G4 constitute a connected component. Similarly, when eachconnection condition is detected with respect to the grids G1, G2, G4,the grids G1, G2, G3, G4 constitute another connected component.

[0038] Now, a face extracting method which applies the object extractingmethod using the motion picture according to the present invention willbe explained.

[0039]FIGS. 3A and 3B illustrate a pair of still image frames having apredetermined time interval, wherein FIG. 3A is a first still imageframe IMAGE(t) at a time (t) and FIG. 3B is a second still image frameIMAGE(t+Δt) after a predetermined time (Δt) elapses from the time (t).

[0040] First, in order to find the motion change of the face inaccordance with the time change, a difference value between the firststill image frame IMAGE(t) and the second still image frame IMAGE(t+Δt)is obtained by each pixel and the resultant values thereof are comparedwith the threshold value which has been previously defined. Accordingly,when the resultant values are greater than the threshold value, adifference image frame can be obtained as shown in FIG. 4.

[0041] On the other hand, FIG. 5 illustrates a skin color image frameobtained from the second still image frame IMAGE(t+Δt) of FIG. 3B. Asdescribed above, the skin color image frame may be obtained from thefirst still image frame, and although the skin color image frameobtained from the first still image frame is applied to the objectextracting method according to the present invention, the result is thesame as a result of the skin color image frame obtained from the secondstill image frame.

[0042] According to the object extracting method of the presentinvention, a logic image frame can be obtained by ANDing the differenceimage frame of FIG. 4 and the skin color image frame of FIG. 5 and thelogic image frame includes the motion change information and the skincolor information of the face. Here, since the logic image frame iscomposed of pixel units, numerous computation processes are required toextract a shape of the face. Thus, the grid process is applied to obtaina grid image frame as shown in FIG. 7. Here, when the 8-directionconnection information is applied to the grid image frame composed ofgrids which are dispersed therein, a connected component correspondingto an area of the face can be generated.

[0043] In FIG. 7, there are seven connected components, and FIG. 8illustrates seven rectangles each defines a minimum area which includeseach of the connected components. Here, when the minimum area isexpressed as a variable RECT[i] wherein i indicates a number of minimumareas, the minimum areas are RECT[1], RECT[2], . . . , RECT[7]. Thus,the conditions of the specific object are compared with each of theminimum areas and the minimum area which satisfies the conditionsthereof can be obtained.

[0044] For instance, the variable RECT[3] of the minimum area iscompared with the conditions of the object as follows.

[0045] (1) A size of the RECT[3]>a threshold value of a size of theminimum area

[0046] (2) A minimum value of a row/column ratio of the RECT[3]<therow/column ratio of the RECT[3]<a maximum value of the row/column ratioof the RECT[3]

[0047] (3) A density of the RECT[3]<a threshold value of a density.

[0048] Here, the threshold value of the size of the face, and theminimum value and the maximum value of the row/column ratio of thevariable RECT[3] are the values which are respectively defined inaccordance with the object. While, the density of the variable RECT[3]is a value of which the density of the connected components is dividedby the area of the variable RECT[3].

[0049] When the above-described method is applied to all of the sevenminimum areas, there are remained several minimum areas which will beface area candidates (not shown).

[0050] The minimum area RECT[3] which has the largest size among theremaining minimum areas is selected as the face area. In the variableRECT[3] of a minimum rectangle which includes connected components thedensity indicates a value of which the number of grids in a row or acolumn of one of the connected components is divided by a length of therow or the column thereof. Since, the minimum area RECT[3] is arectangle, each row or column of the minimum area obtains its densityand each density is compared with a threshold value of the density andthe row or column of which density is smaller than the defined thresholdvalue is deleted, thereby optimizing the minimum area to become theshape of the face.

[0051] As described above, the object extracting method using the motionpicture according to the present invention rapidly and accuratelyextracts the particular object using the information of the motionchange of the object in the motion picture and the color informationwhich define the color of the object.

[0052] Although the human face is taken as the present invention, theobject extracting method of the present invention can be applied to anyobject which has change of its motion and color information thereof, ifthe color information thereof is differently defined. Further, if thereare a plurality of objects to be extracted in the motion picture, anobject which is nearest to a camera, that is the object which has alargest size thereamong, is determined as a specific object. However,when using a templete or a neural network, the specific object can beaccurately and rapidly extracted even though there are a plurality ofobjects in a motion picture.

[0053] It will be apparent to those skilled in the art that variousmodifications and variations can be made in the object extracting methodusing the motion picture of the present invention without departing fromthe spirit or scope of the invention. Thus, it is intended that thepresent invention cover the modifications and variations of thisinvention provided they come within the scope of the appended claims andtheir equivalents.

What is claimed is:
 1. An object extracting method using a motionpicture, said method comprising the steps of: obtaining a differenceimage frame by getting a pair of still image frames having apredetermined time interval from a motion picture and obtaining a colorimage frame which satisfies color information defining a color of aparticular object from one of the still image frames; performing a gridprocessing of a logic image frame which is obtained from the differenceimage frame and the color image frame at a predetermined size andobtaining connected components using direction connection informationand defining minimum areas each includes the connected components;comparing each of the minimum areas with predetermined conditions,thereby selecting the minimum areas which satisfy the conditions asobject area candidates; and extracting an object by selecting andoptimizing an object area candidate which has a largest size among theobject area candidates.
 2. The method of claim 1 , wherein thepredetermined time difference is a time that motion change of theparticular object can be detected in the motion picture.
 3. The methodof claim 1 , wherein the difference image frame only represents an areawhere there is the motion change of the particular object between thetwo still image frames.
 4. The method of claim 1 , wherein theparticular object is a human face.
 5. The method of claim 1 , whereinthe color information is a color which a user arbitrarily defines byadjusting color condition of the object in specific color space.
 6. Themethod of claim 1 , wherein the color information represents a skincolor.
 7. The method of claim 1 , wherein the color image frame isdefined by RGB.
 8. The method of claim 1 , wherein the color image frameis obtained from one of the still image frames.
 9. The method of claim 1, wherein the logic image frame is obtained by ANDing the differenceimage frame and the color image frame.
 10. The method of claim 1 ,wherein the direction connection information indicates a connectionstate of a grid to other grids with respect to eight directions.
 11. Themethod of claim 1 , wherein each of the connected components is composedof grids which are gathered according to the direction connectioninformation.
 12. The method of claim 1 , wherein the minimum area is arectangle which includes the connected components.
 13. The method ofclaim 1 , wherein the predetermined conditions are: (1) A size of avariable RECT[i]>a threshold value of a size of the minimum area; (2) Aminimum value <the row/column ratio of the variable RECT[i]<a maximumvalue; (3) A density of the variable RECT[i]<a threshold value of adensity, wherein the threshold value of the size of the minimum area,and wherein the minimum value and the maximum value of the row/columnratio of the variable RECT[i] are previously defined values inaccordance with the object, and the density of the variable RECT[i] is avalue of which the density of the connected components is divided by anarea size of the variable RECT[i].
 14. The method of claim 1 , whereinthe largest object candidate area is located nearest to a camera. 15.The method of claim 1 , wherein in the step of optimizing the selectedobject area candidate, each row or each column of the object areacandidate obtains its density and each density is compared with thedefined threshold value of the density and the row or column of whichdensity is smaller than the threshold value is deleted, thus beingoptimized to the human face.
 16. The method of claim 15 , wherein thedensity is in the minimum area variable RECT[i] which includes theconnected components the density is a value of which the number of gridsin a row or a column of one of the connected components is divided by alength of the row or the column thereof.
 17. The method of claim 1 ,wherein the step of extracting the object comprises optimizing each ofthe object area candidates and selecting the largest area candidatethereamong.